Repository navigation
[None][feat] Merge back Mistral Large 4 support from feat/mistral-large-4 - #21262
Draft
yizhang-nv wants to merge 20 commits into
Draft
yizhang-nv wants to merge 20 commits into
yizhang-nv wants to merge 20 commits into
Conversation
Add Mistral Large 4 to the PyTorch backend as modeling_v2 targets, with serving support and tests. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Sharabesh Jayaraman <338298191+sharabeshj-nvda@users.noreply.github.com> (cherry picked from commit eb6fc51) Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> Co-authored-by: tburt-nv <195370667+tburt-nv@users.noreply.github.com> (cherry picked from commit 30a9367) Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
…21336) Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> (cherry picked from commit ee7c59a) Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com> (cherry picked from commit 00c184e) Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
mistral-common 1.13.0 requires Python 3.11+, so the PY310 package sanity stage could not install requirements and never built the cp310 wheel. Fall back to mistral-common>=1.10.0 on Python 3.10. Signed-off-by: EmmaQiaoCh <qqiao@nvidia.com>
…eel (#21425) Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
1 task done
…d mapper (#21548) The mistral_large_4 checkpoint format registered the generic HF weight loader and mapper, and HfWeightLoader._requires_lazy_safetensors sniffed every checkpoint directory for a Mistral-native params.json to force lazy SafeTensors for Mistral Large 4. Move both into checkpoints/mistral/: - MistralLarge4WeightLoader subclasses HfWeightLoader, always keeps the checkpoint mmapped, and is the default weight loader of MistralLarge4CheckpointLoader. - MistralLarge4WeightMapper subclasses HfWeightMapper and renames nothing, as before. hf/weight_loader.py and hf/weight_mapper.py return to their state before Mistral Large 4 support was added. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
…of removed paths (#21616) Remove defensive code that Mistral Large 4 support added, and code that only served paths removed earlier (the FP8 target and mistral-common releases before 1.13.0): - Require mistral-common 1.13.0 with one version check where the native config loader recognizes a Mistral Large 4 params.json, and drop the tokenizer and JsonSchema fallbacks for older releases. - Drop the FP8 target's leftovers: the input processor's serves-images check, the NVFP4 condition in routing.serves_images and the refusal of a params.json quantization block (routing rejects it as quantization that no target fits). - Check each chat request rule once: the native renderer no longer rejects chat_template, and the serving extension no longer prechecks reasoning-effort names that mistral-common validates when rendering. - Drop checks that routing or the callers already guarantee: the cache manager's structural checks become one assert; the sharing-length and DSpark layer-order checks, the recurrent DSpark head raise, the attention-DP payloads without spec-decoding fields, the "owns no KEY buffer" raises, the native image count invariants and lm_eval's token id validation go. - State dsa.fp8_kv_cache as the ModelOpt kv_cache_quant_algo in the native config adapter, so ModelConfig and ModelLoader need no Mistral Large 4 branch. Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
chenfeiz0326
added a commit
to chenfeiz0326/TensorRT-LLM
that referenced
this pull request
Nov 11, 2026
… models Mistral Large 4 perf-sanity cases run once every 2 post-merge builds, like the other key models. Its cases (NVIDIA#22056) reach main with NVIDIA#21262 and belong in the _key_models lists. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
…ith the routed experts (#22253) Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
…de on SM107 (#22252) Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
… projections on SM107 (#22251) Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
…ication on SM107 (#22249) Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
… decode on SM107 (#22250) Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com> Signed-off-by: tayuan <51468697+yt0428@users.noreply.github.com>
Collaborator
|
Hi, heads-up: #22286 has merged, so ModelingV2 now lives at Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path. |
…s own decode selector (#22467) Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR merges the Mistral Large 4 support on the
feat/mistral-large-4release branch back intomain.feat/mistral-large-4is the branch published for the Mistral Large 4 launch, and Mistral's launch announcement links to it. It holds one squashed commit,fece87edc4("[None][feat] Add Mistral Large 4 support"), on top ofmainat59eab99c89. Merging it back puts Mistral Large 4 support intomainand into regular TensorRT LLM releases.Note
Placeholder, not ready for review. This draft tracks the merge-back. The branch has not been rebased onto current
mainyet. The PR will be rebased, its scope finalized and CI run before reviewers are requested.Summary of changes
Model support (modeling_v2)
mistralai/Mistral-Large-4-1T-A52B-NVFP4on GB300 (sm_103):mistral_large_4_nvfp4__sm_103__tep4,__dep4and__tp4. The two-node, 8-GPU deployments (TEP8, DEP8, TP8) use the same targets.fp8_ds_mla) KV cache with a dedicated cache manager (tensorrt_llm/_torch/attention/backends/sparse/mistral_large4/).Checkpoint loading
checkpoint_format: mistral_large_4. With it, the model configuration, weights, tokenizer (tekken.jsonthrough mistral-common), chat rendering and image preprocessing all come from the Mistral-format files; the Hugging Face files are not read. The FP8 checkpoint is refused at load time.llm_args.pyinto the checkpoint adapters (tensorrt_llm/_torch/models/checkpoints/dspark_config.py), which also handle a draft embedded in a Mistral-format checkpoint.Serving
mistral_large_4tool parser, themistral_large_4reasoning parser and a serving extension. The extension renders chats with mistral-common and takes the reasoning mode and response format from mistral-common's model settings.xgrammarandllguidancebackends over the Tekken vocabulary. The guided-decodingencoded_vocabnow accepts raw token bytes.store=false, and history tool calls whose arguments are not a JSON object.Runtime and tools
3rdparty/patches/flashmla_release_gil.patch).trtllm-benchreads the checkpoint in the format named byextra_llm_api_options.Dependencies
llguidance0.7.29 → 1.7.6, because 0.7.29 cannot build the Tekken token trie.mistral-common>=1.10.0→mistral-common[opencv]>=1.13.0. 1.13.0 knows the checkpoint's reasoning efforts, and theopencvextra is needed for image input.Documentation
docs/source/deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md, which lists the supported configurations, features and known limitations.Test Coverage
tests/unittest/_torch/modeling_v2/);cpu_only);tests/unittest/_torch/attention/sparse/mistral_large4/);tests/unittest/disaggregated/test_mistral_large4_kv_transfer.py).tests/integration/defs/accuracy/modeling_v2/test_mistral_large4.py:tests/integration/defs/accuracy/test_mistral_large4_disaggregated.py.l0_cpu.yml,l0_gb300.yml,l0_gb300_multi_gpus.ymlandl0_gb300_multi_nodes_node2_gpu8.yml.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
🤖 Generated with Claude Code